Run tests with pytest #963

craigds · 2018-10-02T21:40:38Z

What does this PR do?

https://trac.osgeo.org/gdal/wiki/rfc72_pytest

The intent is to use pytest to run the GDAL test suite.

Large parts of this PR are automated, but manual pieces are in their own separate commits for reviewability. The automated commits are probably too large to review on github.

Notable changes and their implications

tests are now run with cd autotest ; pytest. (The first time you may need to pip install -r requirements.txt to install pytest)
All tests now run in a single process (they were previously forked for each test module). In future it's possible to multi-process them and parallelise, but I avoided doing that in this PR for simplicity. This means that:
- errors during test collection are now loud, and immediately fail the entire test run with a traceback. Previously things like syntax errors in files and errors at module level were easy to miss.
- A single segfault will kill the entire test run dead. (note: This might be handle-able via the xdist plugin, it seems to restart failed worker processes; is that behaviour desirable though? Segfaults are bad ;) )
It's now possible to run individual tests, instead of just entire files. However, tests are not yet independent of each other. So that might cause the tests to behave differently than if you ran the whole module. YMMV.
All tests have been renamed test_<name>. Pytest finds tests by matching names against a regex (this is the default regex).
test_py_scripts.run_py_script was modified to always run the script as a subprocess. The stdout capturing of the original method did strange things with pytest. This change broke some tests that relied on passing files in the /vsimem/ root to scripts, so I've changed those to use the tmp/ root instead.

Changes not included in this PR

I haven't made test functions independent of each other. You still have to run an entire module at a time for most modules. I intend to do this in future but it's out of scope for this PR.
I haven't removed all the global gdaltest.<drivername>_drv variables and replaced them with pytest fixtures. That's probably related to making functions independent, and is also future work.
I haven't done a style cleanup. In particular my automated changes might make code style worse in some respects, and I haven't fretted too much about that. For example, I changed if statements to asserts. Some asserts have turned into really long lines as a result. I think automated style cleanup could easily be done as a separate PR after all this, probably using black.

What are related issues/pull requests?

Convert python test suite to pytest #949
gdal-dev post
The WIP automation code currently lives at https://github.com/craigds/gdal-pytest/

Tasklist

craigds · 2018-10-04T10:42:08Z

⚠️ I'm going to be rebasing & force-pushing this repeatedly until it is merged. The difficulty of merging upstream changes into this is prohibitive; it's much easier to just rebase and re-run the automated changes from scratch.

So, basically, please clone this and have a look at it, just be prepared to force-reset your local copy when I update it 😄

…driver

Pytest truncates expressions from failed asserts, which is a bit counterproductive in CI

… file

craigds · 2018-12-10T23:02:53Z

🎉

rebased it to squash some of the obviously-related commits together, but I'm mostly happy with the history and don't want to spend ages re-ordering it.

@rouault merge when you're happy 😄

craigds · 2018-12-10T23:05:00Z

... unless you think the commit history needs improving in which case I can spend a little longer on it of course.

rouault · 2018-12-10T23:11:26Z

unless you think the commit history needs improving

No, that'll be fine like that. This will reflect well the effort you put in this :-)

craigds · 2018-12-11T00:00:59Z

frustrating OSX CI error; no changes since previous successful build 🤦‍♂️

craigds · 2018-12-11T01:51:53Z

At @schwehr's suggestion I did a quick performance comparison on my vagrant box:

master:
589
690
693
(average 657s)

pytestify:
707
768
759
(average 744s)

So basically this adds 87s on to test runs.

I suppose some performance drop is inevitable but I'm a bit surprised it's as large as that.

craigds · 2018-12-11T01:53:04Z

In the meantime, the OSX CI error went away with a rebuild 👍 ; maybe that one's unrelated to this effort

craigds · 2018-12-11T03:34:54Z

I don't think the increased test time is a problem particularly. I'd prefer to just mitigate it by disabling a few of the slower tests by default:

(output from pytest --durations=10)

137.55s call     gdrivers/wms.py::test_wms_16
127.26s call     gcore/vsis3.py::test_vsis3_read_credentials_ec2_expiration
27.98s call     ogr/ogr_gft.py::test_ogr_gft_ogr2ogr_spatial
17.39s call     ogr/ogr_carto.py::test_ogr_carto_test_ogrsf
16.74s call     ogr/ogr_gft.py::test_ogr_gft_write
16.21s call     gdrivers/mrf.py::test_mrf_lerc_with_huffman
14.00s call     gdrivers/gdalhttp.py::test_http_2
12.88s call     ogr/ogr_gft.py::test_ogr_gft_ogr2ogr_non_spatial
12.30s call     gdrivers/wms.py::test_wms_11
12.19s call     gdrivers/mbtiles.py::test_mbtiles_3

I won't do that here because it just makes the comparison less fair, but happy to do it as a followup PR.

rouault · 2018-12-11T15:34:00Z

@craigds I've rebased my gdalbarn branch on top of master with the pytest changes, and I now get a "The job exceeded the maximum log length, and has been terminated." on the trusty_clang config : https://travis-ci.org/rouault/gdal/jobs/466516798 . This is apparently really related to the log length, and not the time out
Any idea how to fix that ?

rcoup · 2018-12-11T15:45:28Z

Yeah, I think maybe we take away the pytest -vv ... in script.sh and swap it for -ra, so it won't print anything at all for passed tests, and still have verbose details for failures + errors. But there's a confusing set of options there, maybe @craigds can confirm?

rouault · 2018-12-11T16:28:32Z

At the same time, if we don't print anything for too long, Travis will kill us because of this. I'm not clear why the pytest output doesn't please Travis. Perhaps each line is just slightly longer than before, but the number of lines should be similar.

rouault · 2018-12-11T23:01:15Z

Using -ra did the job.

craigds · 2018-12-11T23:53:27Z

@rouault One less drastic thing we could do is add -o console_output_style=classic to pytest invocations, which would supress the percentage progress indicator and all the whitespace padding to the right of the test names. I already did that in gcc52_stdcpp14_sanitize environment; can't remember why

craigds · 2019-08-03T21:56:30Z

I gave a talk about this effort yesterday at Pycon AU.

Description: https://2019.pycon-au.org/talks/how-i-migrated-a-huge-oss-project-to-use-pytest
Video: https://www.youtube.com/watch?v=SLbOWiz4KSs

rouault · 2019-08-04T00:04:34Z

Thanks for the link, and thanks again for the incredible job you did ! I enjoy so much being able to write tests in a more compact way.

craigds force-pushed the pytestify branch 2 times, most recently from 92dd2fb to 2537360 Compare October 4, 2018 10:32

craigds force-pushed the pytestify branch 7 times, most recently from 4a4a190 to 4d7963e Compare October 11, 2018 22:12

craigds force-pushed the pytestify branch 14 times, most recently from cae613f to 1c9b78e Compare October 19, 2018 11:17

craigds force-pushed the pytestify branch 6 times, most recently from 50d0e31 to bc96bde Compare October 24, 2018 08:33

craigds added 11 commits December 11, 2018 12:01

Fix sqlite tests when compiled without spatialite

d87dc34

Run tests in per-module subprocesses in the address sanitizer env

ada1eb4

Remove run_tests() function, remove last usages of it

a0d6a07

MinGW build tweaks

b65c5d1

Add some basic docs. Remove pin for pytest; v4 should work ok

7e62d38

Remove a number of gdaltest_list mentions, no longer needed/used

33b7a95

Update appveyor/windows build to use pytest

41e15ac

Include but skip dods/gmlas tests. Include fgdb stress tests if fgdb …

10d7106

…driver

Add more verbosity to test output in CI

6c3d9aa

Pytest truncates expressions from failed asserts, which is a bit counterproductive in CI

Make vrt_read_29 test not care whether gdal.Translate closes the dest…

b54e910

… file

Tweak the IDs in parametrized tests to be more useful

46e92b7

craigds force-pushed the pytestify branch from 8db599e to 46e92b7 Compare December 10, 2018 23:01

Fix bug in vagrantfile script to install pip

b2453ed

rouault merged commit 59d0840 into OSGeo:master Dec 11, 2018

craigds mentioned this pull request Jan 10, 2019

Tests: Some test cleanup, mostly boilerplate removal #1194

Closed

3 tasks

QuLogic mentioned this pull request Feb 8, 2021

autotest: Parameterize alg tests. #3445

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests with pytest #963

Run tests with pytest #963

craigds commented Oct 2, 2018 •

edited

Loading

craigds commented Oct 4, 2018 •

edited

Loading

craigds commented Dec 10, 2018 •

edited

Loading

craigds commented Dec 10, 2018

rouault commented Dec 10, 2018

craigds commented Dec 11, 2018

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Dec 11, 2018

rouault commented Dec 11, 2018

rcoup commented Dec 11, 2018

rouault commented Dec 11, 2018

rouault commented Dec 11, 2018

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Aug 3, 2019 •

edited

Loading

rouault commented Aug 4, 2019

Run tests with pytest #963

Run tests with pytest #963

Conversation

craigds commented Oct 2, 2018 • edited Loading

What does this PR do?

Notable changes and their implications

Changes not included in this PR

What are related issues/pull requests?

Tasklist

craigds commented Oct 4, 2018 • edited Loading

craigds commented Dec 10, 2018 • edited Loading

craigds commented Dec 10, 2018

rouault commented Dec 10, 2018

craigds commented Dec 11, 2018

craigds commented Dec 11, 2018 • edited Loading

craigds commented Dec 11, 2018 • edited Loading

craigds commented Dec 11, 2018

rouault commented Dec 11, 2018

rcoup commented Dec 11, 2018

rouault commented Dec 11, 2018

rouault commented Dec 11, 2018

craigds commented Dec 11, 2018 • edited Loading

craigds commented Aug 3, 2019 • edited Loading

rouault commented Aug 4, 2019

craigds commented Oct 2, 2018 •

edited

Loading

craigds commented Oct 4, 2018 •

edited

Loading

craigds commented Dec 10, 2018 •

edited

Loading

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Dec 11, 2018 •

edited

Loading

craigds commented Aug 3, 2019 •

edited

Loading